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I.  Objectives 


n  Overall  Objective 

■'  Effective  collaboration  through  improved  collaborative  critical  thinking 
(CCT) 

n  Objectives  for  this  year 
!■  Update  /  revise  /  improve 

•  CCT  tool  (CENTER) 

•  CCT  Training 

f  Perform  lab-based  validation  study 

•  USF 

^  Prepare  for  and  perform  Field  validation  study 

•  NWCDDDLab 

I-  Prepare  for  fielding  of  tool  (non  ODD) 

•  NETWARCOM 

•  FORCEnet 
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II.  Final  Products 


*  CCT  Theory  and  Measures 

■  CCT  Collaboration  Tool  (CENTER) 

*  CCT  Training 


Potential  impact 

f  Both  process  and  products 

•  Improved  collaboration 

•  Better  team  decisions 

I'  Measures  of  CCT 
Applications 

I'  Any  distributed,  synchronous  team  working  together 
to  find  a  solution. 
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In  the  beginning,  there  was  theory... 


r*  Information  age  warfare 

■  Teams  are  distributed,  ad  hoc,  multi-disciplinary, 
mission-critical 

I  Teams  require  coordination  &  collaboration 

•  Manage  forces  &  information 

•  Achieve  effects 

I  Supporting  coordination  &  collaboration  requires 
measurement 

['  Coordination  &  collaboration  processes  can  be 
measured 


^  Alberts,  Garstka, 
Hayes,  and 
Signori  (2001) 

!■  Letskyetal. 
(2003) 


S'  Macmillan,  et  al., 
2001 

S'  Miller,  Price,  Entin, 
&  Rubineau,  2001 

S'  Moon,  et  al.,  2000 


I'  Collaboration  often  involves  critical  thinking.  For 
individuals,  CT... 

■  Is  found  in  transcripts  of  planning 
»  Can  be  trained 

■  Improves  mission  performance  in  Air  Defense 
scenarios 


Cohen,  Freeman, 
and  Thompson, 
1998 

Cohen  and 
Freeman,  1997 
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Contribution  to  collaboration  technology 

B  Expand  (individual)  critical  thinking  to  (team) 
collaboration 

B  Collaborative  critical  thinking  is  teamwork  behavior 
that  identifies  and  reduces  uncertainty  concerning 
technical  problems  and  team  process. 

m  Research-based  tool 

»'  Collaboration 

I'  Team  processes 

I'  Critical  thinking 

ii  Tool  that  is  pro-active  and  prescriptive  to  improve 
team  collaboration  in  real-time 
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(Fit  to  Collaboration  Model) 


Collaboration  Drivers,  Enablers,  Processes,  Products  &  Effects 
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CCT  Theory 


Collaboration  Driv 


Collaboration  Drr 

Mission  characteristics 
Resource  &  responsibility  distl 
Expertise  distribution 


Col 


Collaboration 

Processes 


Effects  of  Collaboration 

Effective  allocation  of  assets 
Unity  of  intent 
Coordination 
Mission  Effects 


Products  of  Collaboration 

Assessments  &  SA 
Plans 

Team  process  refinements 
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CCT  Theory 


Effects  of  Collaboration 

Effective  allocation  of  assets 
Unity  of  intent 
Coordination 
Mission  Effects 
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A  Definition 


■  Interactions''  to 

•  mon/for  for  uncertainty, 

•  assess  the  risk  it  presents  &  the  opportunity  to  address  it, 

•  conduct  critiques  that  identify  sources  of  uncertainty,  and 

•  devise  actions  that  reduce  uncertainty  &  risks  or  insure  against 

■  Effects  on 

•  the  mission  at  hand 

•  more  accurate  &  precise  assessments 

•  more  effective  &  robust  plans 

•  team  processes  for  executing  that  mission 
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CCT  Behaviors 


monitoring  interactions 

■■  that  alert  other  team  members  to  the  existence  of  uncertainty 

t“  assessment  interactions 

!■  in  which  team  members  evaluate  the  opportunity  (e.g.,  available  time)  and 
need  (e.g.,  priority  or  stakes)  to  resolve  the  uncertainty 

f"  critiquing  interactions  in  which  team  members 

►  identify  gaps, 

►  untested  assumptions,  & 

►  conflicting  interpretations 

^  actions  reduce  uncertainty,  at  best,  or  confirm  that  it  cannot  be 
immediately  reduced. 
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CCT  Objects 


ti  Team  products 
■'  Assessments 
■■  Plans 

t*  Team  processes 

■'  Goals  -  Common  intent,  Complementary  objectives 
■'  Plans  - 

•  Task  allocation  &  schedules 

•  Asset  allocation 

I'  Tasks  -  Task  execution  strategies 
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Measuring  CCT:  Objectives 


»  Help  team  leaders  &  team  members 

o  Monitor  for  the  need  to  apply  CCT  to  products  & 
processes 

ii  Assess  whether  to  engage  in  CCT 
»'  Critique  weaknesses  in  products  &  processes 
I'  Take  action  to  strengthen  them 

»•  Assess  the  effects  of  CCT  on 
The  mission 
■■  The  team 
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CCT  Measures:  Products  &  Processes 


Objects  of  CCT 

Team  products 

Team  processes 

CCT  Behavior 

Assessment  COA/Pian 

Goais  Pians  Tasks 

Monitoring 

Measures 

Assessment 

Critiquing  for  gaps 

Critiquing  for  assumptions 

Critiquing  for  ambiguity 

Action 

Team  members  rate  status  of  products  &  processes  wrt  CCT  behaviors  during 
mission  execution 

Short  to  minimize  distraction  from  tasks 
Customized  to  the  mission  to  enhance  relevancy 
Time  sensitive  to  enhance  relevancy 

Numerically  scaled  (6=Agree  ...  1=Disagree,  Don’t  know)  to  facilitate  summarization 
Optional  text  comments  to  deepen  meaning 
Present  to  team  leader 
^  Aggregated  quantitative  results 
I'  Comments 
■'  Guidance 


I- 
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CCT  Measures:  Products  &  Processes 


Objects  of  CCT 

Team  products 

Team  processes 

CCT  Behavior 

Assessment 

COA/Pian 

Goais 

Pians 

Tasks 

Monitoring 

A1 

T1 

Assessment 

C2 

Critiquing  for  gaps 

T3 

Critiquing  for  assumptions 

P4 

Critiquing  for  ambiguity 

T5 

Action 

A6 

G6 

I-  A1 :  The  team's  assessment  [of 


r> 


J  is  correct. 


A6:  The  team  is  taking  actions  to  resolve  problems  with  the  assessment 
[concerning _ ] 

C2:  The  team  has  time  to  critique  and  refine  the  plan  [regarding _ ] 

P4:  The  team  has  identified  key  assumptions  that  have  yet  to  be  tested 
concerning  its  strategy  [for _ ]. 

G6:  The  actions  of  team  members  are  consistent  with  the  mission  goals 
[concerning _ ]. 

T 1 :  Team  members  seek  feedback  on  their  tasks  [concerning _ ]. 

T3:  The  team  is  completing  all  tasks  [concerning _ ]. 


T5:  Team  members  seek  to  resolve  ambiguity  i 
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CCT  Measures:  Feedback 


c  Probe:  The  team  has  considered  plausible,  alternative  plans. 

c  High  agreement  (mean  rating),  little  dissent  (low  variability  in 
ratings) 

»  Consider  how  much  effort  was  expended  considering  alternative 
plans. 

p  If  little  effort  was  expended,  team  members  may  be  in  agreement 
concerning  a  hastily  generated  and  incorrect  plan.  Perhaps  team 
members  should  expend  more  effort  considering  alternative  plans.  Ask 
them,  “Have  you  exhausted  all  plausible  alternative  plans?” 

p  If  considerable  effort  was  expended,  then  ask:  “Do  you  have  more 
time  to  consider  alternative  plans?”  If  yes,  then  ask:  “Are  there  any 
plausible,  alternative  plans  that  you  overlooked?”  Probe  for  lone 
dissenters  and  check  in  with  the  dissenter. 

c  Average  agreement  and  lots  of  variability  in  responses 

«  If  little  effort  was  expended,  then  suggest,  “Spend  more  time  working 
on  this  task  to  identify  alternative  plans.” 

■  If  considerable  effort  has  been  expended,  ask,  “Do  you  have  more 
time  to  consider  alternative  plans?”  If  yes,  then  team  members  should 
expend  more  effort  to  consider  plausible  alternative  plans 
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CCT  Measures:  Effects 


Mnemonic 

Measure 

Example 

Who 

The  fit  of  team  members  to 
mission  tasks 

Team  optimization  (Levchuk,  etal.,  1997- 
2003) 

What 

Situational  awareness 

Situation  reconstruction  (Endsley  & 

Garland,  2000);  Relational  knowledge 
(Cooke,  et  al.,  2000) 

Where 

Coordination  in  space 

n.a. 

When 

Synchronization 

Asset  synchronization  (Stacy,  Freeman,  et 
al.,  2005;  Levchuk,  et  al.,  1997-2003) 

Why 

Unity  of  intent 

tbd...(Aptima,  Cooke) 

How 

The  fit  of  tactics  (tasks)  to 
objectives:  simplicity, 
power,  observability, 
vulnerability 

Power  appropriateness  (Levchuk,  et  al., 
1997-2003):  tbd 
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CCT  Technology:  CENTER 
^Collaboration  for  ENhanced  TEam  Reasoning 


Probes  re: 
problem 
&  process 


Ratings  & 
Comments 


University  of 
South  Fiorida 


NAVi^AI  R 


Apt 

J-  u,,r 


inia 

Human  Cantered  Engintannig 


CENTER 

for  a  Team  Facilitator  or  a  Leader’s  Aid 


p  Facilitator  configures  CENTER: 
Customize  CENTER  questions  to 
the  mission  at  hand: 

■I  Problem:  Is  there  a  plausible 
assessment  of  enemy  intent  that 
the  team  has  not  considered? 

■I  Process:  Are  your  team 
members  appropriately  tasked 

to  complete  the  assessment  in 
two  hours? 


p  CENTER  automatically  queries 
team  members 
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CENTER 
for  Team  Members 


f  CENTER  pops  a  question  in 
small  “probe”  window  atop 
current  application. 

ii  Member 

»'  Responds  with  rating 
I'  Optionally  comments 
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CENTER 

for  a  Team  Facilitator  or  a  Leader’s  Aid 


ft  CENTER  compiles  team 
members’  responses 

I'  CENTER  presents  facilitator 
with 

I'  Summary  of  ratings  for  at-a- 
glance  assessment  of  team 

t>  Comments  by  members 

■>  Suggested  actions 

i>  Facilitator  adjusts  the  team’s 
course 
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“Synergistic”  Technology 


*  There  are  distributed  collaboration  platforms 
available 

*>  Infrastructure  from  many  vendors,  for  example 

•  Microsoft  (Sharepoint) 

•  IBM  (Lotus  Notes) 

•  Groove  Networks 

•  Sitescape 

K  The  infrastructure  targets  convenience  and  speed  in 
collaboration. 

•  The  CENTER  tool 

*  Operates  “on  top  of  existing  infrastructure. 

Supplements  those  offerings  by  targeting  the  quality 
of  collaboration. 
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Benefits 


f'  Collaboration  tools  (email,  telecons,  Word  versioning)  help  create  products 

■  CENTER  helps  team  evaluate  products  &  process 
r  Training  helps  teams  learn  new  processes 

■  CENTER  ensures  they  use  new  processes 

t-  Group  decision-ware  (voting  tools)  replace  standard  tools 

■  CENTER  complements  standard  tools 

t  CENTER 

I  Python  programming  language. 

•  An  interpreted,  interactive,  object-oriented  programming  language 

•  Comparable  to  TCL,  Perl,  Scheme,  or  Java. 

•  Can  run  in  any  operating  system. 

■  The  CCT  tool  has  a  client/server  architecture  and  uses  MySQL  for  the  back-end 
database. 

■  The  tool  is  designed  to  run  over  the  Internet. 

■  The  users  of  the  tool  do  not  have  to  be  using  the  same  operating  system  when 
the  tool  is  running. 

•  i.e  some  users  can  be  in  Windows  and  some  can  be  in  Linux  and  still  communicate  and 
pass  information  seamlessly. 
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Aptima*  Current  state  and  future  vision:  Center 
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Tool 


Designed  and  impiemented 

Designed  and  yet  to  be 
implemented 

To  be  designed  and  impiemented 

Participant 

Unobtrusive  pop-up  window 

Radio  buttons  with  anchors  for 
rating 

Questions  are  on  a  timer;  they 
blank  out  and  the  window 
minimizes  when  time  is  up 

Instead  of  blanking  out,  the 
window  wiii  say  “waiting  for  the 
next  probe” 

There  will  be  a  count  down  until 
the  window  closes 

Comment  capabiiity  (currently 
“commented  out”) 

Facilitator  / 
Authoring 

Click  and  send  probes 

“Auto  send”  probes  based  on 
timer 

Real-time  resuits  from  probe 

Feedback  based  on  participant’s 
responses  to  probes 

Ability  to  review  meeting  with 
screen  shots  by  specifying  time 
period 

Real-time  probe  authoring 

Create  authoring  mode 

“Roll  over”  graph  labels 

Probe  organization  scheme  with 
more  probes  “in  storage” 

Other 

Linux  instailation  package 

Windows  executabie  instaliation 
package 

Test  plan  (begin  with  fictitious 
(i.e.,  programmed)  participants; 
then  do  with  reai  people) 

Capabilities  for  spatiaiiy 
distributed  team 

Truly  asynchronous  capabilities 
(separation  in  time  and  space) 

Aptima*  Current  state  and  future  vision:  Center 

Human  Cantered  Engin«annjB 

Tool 


Designed  and  impiemented 

Designed  and  yet  to  be 
implemented 

To  be  designed  and  impiemented 

Participant 

Unobtrusive  pop-up  window 

Radio  buttons  with  anchors  for 
rating 

Questions  are  on  a  timer;  they 
blank  out  and  the  window 
minimizes  when  time  is  up 

Instead  of  blanking  out,  the 
window  wiii  say  “waiting  for  the 
next  probe” 

There  will  be  a  count  down  until 
the  window  closes 

Comment  capabiiity  (currently 
“commented  out”) 
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Tool 


Designed  and  implemented 

Designed  and  yet  to  be 
implemented 

To  be  designed  and  implemented 

Facilitator  / 
Authoring 

Click  and  send  probes 

“Auto  send”  probes  based  on 
timer 

Real-time  probe  authoring 

Create  authoring  mode 

Real-time  results  from  probe 

Feedback  based  on  participant’s 
responses  to  probes 

Ability  to  review  meeting  with 
screen  shots  by  specifying  time 
period 

“Roll  over”  graph  labels 

Probe  organization  scheme  with 
more  probes  “in  storage” 
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Tool 


Facilitator  / 
Authoring 


Designed  and  implemented 

Designed  and  yet  to  be 
implemented 

To  be  designed  and  implemented 

Click  and  send  probes 

“Auto  send”  probes  based  on 
timer 

Reai-time  probe  authoring 

Create  authoring  mode 

probes  i|  Graphs  |  Comments 


Select  a  probe  from  the  following  ist,  then  click  Send 
Probes 

In  your  opinion,  is  the  team's  assessment  o1  the  current  situation  correct? 

Does  your  team  have  all  of  the  critical  infortnation  needed  to  solve  the  problem? 

■  Has  the  team  acidressedthe  plausible  alternatives  tor  solving  ihe  problem? 

In  your  opinion,  are  the  team  members  worhing  toward  the  same  goal? 

To  what  extent  have  the  advantages  and  disadvantages  of  the  solution  been  discussed? 
In  your  opinion,  have  all  feasible  solutions  been  considered? 

How  realistic  is  the  time  ine  for  the  plan? 

How  appropriately  is  responsibilitv  allocated  among  team  members? 

■  How  much  are  team  members  communicating  about  the  task  at  hand? 

How  auccessful  have  team  members  been  with  their  roles? 


Send 


New  probe:  [ 


Elapsed  Time:  |00:00:00 

Probe  Response  Tirieoot:  | 

(seconds)  Send 
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Tool 


Aptima*  Current  state  and  future  vision:  Center 

Human  ^  Cantered  Engin«annjB 

Tool 


Designed  and  implemented 

Designed  and  yet  to  be 
implemented 

To  be  designed  and  implemented 

Facilitator  / 
Authoring 

Click  and  send  probes 

“Auto  send”  probes  based  on 
timer 

Real-time  probe  authoring 

Create  authoring  mode 

Real-time  results  from  probe 

/ 

Feedback  based  on  participant’s 
responses  to  probes 

Ability  to  review  meeting  with 
screen  shots  by  specifying  time 
period 

/  1 

“Roll  over”  graph  labels 

_ / 

Probe  organization  scheme  with 
more  probes  “in  storage” 

Probe:  The  team  has  the  information  it  needs  to  plan. 

Analyses  reveal:  Mean  is  average  and  Variance  is  high 

Feedback  to  facilitator:  If  lots  of  time  has  been  spent,  seek  to 
understand  why  there  is  so  little  consensus.  Perhaps  team  members 
need  to  revisit  prior  decisions  and  then  revisit  the  current  problem. 
Ask,  do  you  understand  the  information? 
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Tool 


Designed  and  implemented 

Designed  and  yet  to  be 
implemented 

To  be  designed  and  implemented 

Facilitator  / 
Authoring 

Click  and  send  probes 

“Auto  send”  probes  based  on 
timer 

Real-time  probe  authoring 

Create  authoring  mode 

Real-time  results  from  probe 

Feedback  based  on  participant’s 
responses  to  probes 

Ability  to  review  meeting  with 
screen  shots  by  specifying  time 
period 

“Roll  over”  graph  labels 

Probe  organization  scheme  with 
more  probes  “in  storage” 

Probeg  1 1  Graphg 

Comments 

*  ....  .  ■■-■■■! 

1  Data  view  [aII  probes  -  Hi,  Low,  Average 

The  team  has  the 
information  it  needs  to 
plan. 
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Aptima*  Current  state  and  future  vision:  Center 

Human  Cantered  Engm«annjB 

Tool 


Designed  and  implemented 

Designed  and  yet  to  be 
implemented 

To  be  designed  and  implemented 

other 

Linux  installation  package 

Windows  executable  installation 
package 

Test  plan  (begin  with  fictitious 
(i.e.,  programmed)  participants; 
then  do  with  real  people) 

Capabilities  for  spatially 
distributed  team 

Truly  asynchronous  capabilities 
(separation  in  time  and  space) 

Current  state  and  future  vision:  Training  and 

Human  ^  Cantered  Engintannig  w 

Validation 


Completed 

Designed  and  in  progress 

To  be  designed  (option  task) 

Development 

Studies  to  better  understand  CCT 

Study  to  investigate  possible 
cognitive  and  affective 
components  of  CCT  for  training 
development 

CCT  training  and  facilitator 
training 

improve  CCT  training  based  on 
results  of  experiment 

improve  CCT  training  to  make 
more  appropriate  for  military  / 
expert  teams 

Usability 

Usability  studies  with  pop-up 
window 

initiai  usabiiity  studies  with 
facilitator  window 

Lab 

Validation 

Study  to  validate  CCT  probes 

Initial  validation  evidence  for 
training 

Initial  validation  evidence  for  tool 

Field 

validation 

1  lOA 

Field  validity  evidence  (Option 
task) 

anu  usG 

Use  in  field  setting 
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III.  Experiments 


I"  Completed  Studies 

■'  Studies  to  better  understand  CCT 

•  We  were  able  to  get  examples  of  CCT  to  guide  definition  and 
framework  development 

■'  Studies  to  test  possible  impact  of  cognition  and  affect  to  guide 
training  development 

•  Neither  the  cognitive  nor  the  affective  training  made  an  impact  in 
CCT  performance;  thus,  they  were  not  included  in  the  training 

■'  Usability  studies 

•  Guided  improvements  to  the  user  pop-up  window  and  the  facilitator 
workspace 

■'  Initial  study  to  validate  critical  thinking  probes 

•  Probes  were  at  the  correct  level  for  the  target  population 

•  Participants  felt  that  probes  helped  them  execute  planning  tasks 

I'  Validation  Study  at  USF 

•  Behaviorally-based  CCT  training  resulted  in  improved  task 
performance 
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Most  Recent  Experiment 


Still  on-going  at  USF 
Objective: 

»  Collect  validity  evidence  for  CENTER  tool  and  training 


Method: 


Each  of  2  team  mem 

•  No  training 

•  CCT  Training 

•  Or  control  traini. 
Teams  plan  for  coni^ 

•  With  CENTER  tool 

•  Or  not 

Teams  execute  Sear 


•Content 

•Content 

•Basic  Steps  to  Survival 

•S  -  Size  up  the  Situation 

•U  -  Use  all  Your  Senses,  Undue  Haste  Makes 

Waste 

•R  -  Remember  Where  You  Are 
•V  -  Vanguish  Fear  and  Panic 
•I  -  I  mprovise 
•V  -  Value  Living 
•A  -  Act  Like  the  Natives 

•L  -  Live  by  Your  Wits,  but  for  Now,  Learn  Basic 
•[  Skills 


tc  •Process  (“hands-on"  task) 


•t  •Content 
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Approach 


»'  Experimental  Design: 

I'  5  conditions 


Conditions 

No  CCT  probes 

CCT  probes 

No  training 

X 

Survival  training 
(controi) 

X 

X 

CCT  training 

X 

X 

I'  Performance  measures  collected  through  the  DDD 
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Analyses 


i  Measures 

■  Used  by  tool  to  gauge  CCT 

•  Probes 

•  Analyses  done  in  real  time 

i  Used  to  validate  CCT  training 

•  DDD  Performance 

•  Number  of  clues  /  UGS  processed  correctly 

•  Time  to  find  lost  party  /  antenna 

•  Debrief  Questions 

•  Did  the  probes  make  you  think  differently  about  the  problems  or  how 
the  team  needed  to  perform? 

•  How  influential  were  the  probes  in  your  thinking? 


■  Analyses 

I  Evaluate  impact  of  training 

t  Test  hypothesis  that  teams  participating  in  CCT  training  perform  better 
on  the  DDD 

i  Hierarchical  analyses  to  investigate  contributions  of  individual 
characteristics 

N  A\/^A  I  R 
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Findings* 


i  Both  CCT  Training  and  the  Survival  training  improved  the 
team  performance  over  no  training 

i  The  effects  of  the  CCT  Training  and  the  Survival  training  on 
performance  was  statistically  the  same. 

■  Two  different  methods  to  increase  team  performance 

•  One  focused  more  on  critical  thinking  and  context  free  decision 
making 

•  One  focused  on  decision  making  in  a  naturalistic  environment 
similar  to  the  actual  task  environment 

i  Some  individual  differences  traits  are  relevant  to  the  group 
dynamic: 

■  CCT  is  enhanced  with 

•  Increasing  agreeableness  of  team  members 

•  Increasing  assertiveness  of  team  members, 

•  IQ 


*Note:  CENTER  tool  validity  is  in  the  process  of  being  tested 
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IV.  Future  Validation 


»  USF  CENTER  Tool  validation 

Teams  plan  for  ODD  computer  simulation 

•  With  CENTER  tool 

•  Or  not 

i>  Field 

f  Field  validation  study 

•  NWCDDDLab 

I'  Non-DDD  fielding  of  tool 

.  NETWARCOM 

•  FORCEnet 
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contributions 

■>  Society  for  Industrial  and  Organizational 
Psychologists  2005  Academic  Practitioner  Forum 

ii  Freeman,  J.  and  Hess,  K.  (2003).  Collaborative 
critical  thinking.  Proceedings  of  the  8th  International 
Command  and  Control  Research  and  Technology 
Symposium.  Washington,  DC. 

II  Freeman,  J.  and  Serfaty,  J.  (2002).  Team  critical 
thinking.  Proceedings  of  the  7th  International 
Command  and  Control  Research  and  Technology 
Symposium.  Monterey,  CA. 
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VI.  Lessons  Learned 


»  When  working  with  others  you  take  the  good  with  the 
bad 

I'  Sometimes  participants  learn  from  the  “control” 
training 

»  Nothing  ever  goes  the  way  it  is  planned. 
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Project  Summary 

«  Title:  Collaborative  Critical  Thinking  (CCT) 

Jared  Freeman,  Ph.D.,  P.l. 
i  Aptima,  1030  15*^  Street  NW,  Washington,  DC  20005 
I  202-842-1548x316 
»  freeman@.aptima.conn 
I  Kathleen  Hess,  Ph.D.,  P.M. 

i  Aptima,  12  Gill  St.  Suite  1400,  Woburn,  MA  01801 
i  781-935-3966X219 
i  khess@aptima.com 
r  Objectives 

»  Better  understand  CCT 

i  Develop  validated  training  and  tools  to  improve  CCT 
i  Improve  the  process  and  products  of  collaboration  through  improved  CCT 
I  Research  Questions 

i  Can  CCT  behaviors  and  their  effects  be  reliably  measured  in  a  semi-  or  fully-automated  fashion? 
I  Can  we  promote  CCT  behaviors  with  training  and  job  aids? 

■  Does  improved  CCT  result  in  improved  collaboration? 
i  Can  we  field  the  CENTER  tool? 

I  Project  Status 

i  Initial  validation  studies  have  been  run 
1  Tool  and  training  are  being  revised  based  on  the  study  results 
■"  Field  validation  studies  are  being  planned 
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Supplemental  Material 
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Sign  on  dialog 


Note;  When  signing  on  to 
the  CCT  Tool  the 
participants  are  already 
logged  onto  the 
collaborative  application. 


Remember;  during  the  work  session  the  critical  thinking 
assessment  tool  will  refer  to  you  as 


/ 
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Coordinator’s  Configuration  Interfaces 


i  Collaborator  -  Critical  Thinking  Assessment  Set  Up 


xj 


:  Work  Session  :j  Probes)  Participants ) 


Name; 

Date; 

Location; 

Obiective; 

Notes; 


I - 

1  6/10/2003  ^  Time:  |aM  |AM 


r 


Save 


Configure  (clockwise) 

■  Work  session 

■  Probes  &  schedule 

■  Participants 
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i.  Collaborator  -  Critical  Thinking  Assessment  Set  Up 


Work  Session  L.,Pfo!^j|  Participants) 
Compose  probe; 


x] 


Add  ) 


Probe  # 

Probe  content 

Presentation 

Time 

1 

Probe  number  1 

15 

2 

Probe  number  2 

30 

3 

Probe  numbers 

45 

Import ...  ) 


Probe  presentation  sequence; 


5 

15 

30 

45 

60 

Time  Tom  start  > 

1 

2 

3 

Probe  # 

"‘To  diange  the  presentation  time  of  a  probe  slide  it  to  the  desired  time 


I  Collaborator  -  Critical  Thinking  Assessment  Set  Up 


xj 


Work  Session  )  Probes 


Participants  in  the  critical  thinking  assessment; 


1“  Anastasi  Donna 
I”  Bailey  Adam 
1“  Baker  Keith 
1“  Chopra  Kari 
1“  Might  Heather 
1“  LevchukYuri 
r”  Miller  Diane 

"  Not  all  participants  in  the  work  session  need  to  participate  in  the 
critical  thinking  assessment 


Insert  participant  list ...  | 


Coordinator  of  critical 
thinking  assessment; 

Leader  of  work  session; 
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Facilitator  Screenshot 


Univt 

Souti 


".ii:>napshoF’~of  teattf 


I  your  opinion,  is  the  team's  assessment  of  your  squad's  current  limitations  correct? 


Does  your  team  have  all  of  the  critical  information  needed  for  developing  orders? 

Has  the  team  addressed  the  plausible  alternatives  for  the  proposed  orders? 

Is  the  team  pursuing  the  information  critical  to  understanding  the  health  of  the  lieutenant? 
Is  the  team  pursuing  the  information  critical  to  address  the  needs  of  squads  2  &  3? 

Do  you  feel  that  the  orders  are  correct  with  respect  to  the  safety  of  squads  2  &  3? 

Do  you  feel  that  the  orders  are  correct  with  respect  to  the  health  of  the  li  eutenant? 

In  your  opinion,  are  the  team  members  working  toward  developing  the  same  orders? 

Send 


d 


New  probe: 


Add  Add  &  Send 


Aptinia 

Hi  I  man  r  flnnta 
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Facilitator  Screenshot 


Critical  Thinking  Assessment  Tool 


Participants 

Probe  Result 

Comments 

Participant  One 

Participant  Two 

7  n 

G  - 

5  - 

4  - 

3  - 

2  - 

1  - 

Current  Probe 

1 )  These  are  comments  for  thi  s  probe . 

2)  These  are  more  comments  from  the  other  participant. 

i Probes!  Graphs  Comments 


Select  a  pro^ 

E£ 


(lowing  list,  then  click  Send 


I  your  opinion,  is  the  team's  assessment  of  your  squad's  current  limitations  correct? 


Does  your  team  have  all  of  the  critical  information  needed  for  developing  orders? 

Has  the  team  addressed  the  plausible  alternatives  for  the  proposed  orders? 

Is  the  team  pursuing  the  information  critical  to  understanding  the  health  of  the  lieutenant? 
Is  the  team  pursuing  the  information  critical  to  address  the  needs  of  squads  2  &  3? 

Do  you  feel  that  the  orders  are  correct  with  respect  to  the  safety  of  squads  2  &  3? 

Do  you  feel  that  the  orders  are  correct  with  respect  to  the  health  of  the  li  eutenant? 

In  your  opinion,  are  the  team  members  working  toward  developing  the  same  orders? 


probes  available  to  seiioi 


Univt 

Souti 
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Facilitator  Screenshot 


Critical  Thinking  Assessment  Tool 


Participants 
Participant  One 
Participant  Two 


Probe  Result 


Comments 


1 )  These  are  comments  for  thi  s  probe . 

2)  These  are  more  comments  from  the  other  participant. 


Probes  i  Graphs  Comments  ! 


Select  a  probe  from  the  following  list,  then  click  Send 
Probes 


I  your  opinion,  is  the  team's  assessment  of  your  squad's  current  limitations  correct? 


Does  your  team  have  all  of  the  critical  information  needed  for  developing  orders? 

Has  the  team  addressed  the  plausible  alternatives  for  the  proposed  orders? 

Is  the  team  pursuing  the  information  critical  to  understanding  the  health  of  the  lieutenant? 
Is  the  team  pursuing  the  information  critical  to  address  the  needs  of  squads  2  &  3? 

Do  you  feel  that  the  orders  are  correct  with  respect  to  the  safety  of  squads  2  &  3? 

Do  you  feel  that  the  orders  are  correct  with  respect  to  the  health  of  the  li  eutenant? 

In  your  opinion^i;gilaaJ®«wifwmlier?WWrn5TowafT3eve^ 


Facilitator  Screenshot 


ProbeSj  Graphs  ^  Comments 


Select  a  probe  from  the  following  list,  then  click  Send 
Probes 


In  your  opinion,  is  the  team's  assessment  of  the  current  situation  correct? 

Does  your  team  have  all  of  the  critical  information  needed  to  solve  the  problem? 

Has  the  team  addressed  the  plausible  alternatives  for  solving  the  problem? 

In  your  opinion,  are  the  team  members  working  toward  the  same  goal? 

To  what  extent  have  the  advantages  and  disadvantages  of  the  solution  been  discussed? 
In  your  opinion,  have  all  feasible  solutions  been  considered? 

How  realistic  is  the  time  line  for  the  plan? 

How  appropriately  is  responsibility  allocated  among  team  members? 

How  much  are  team  members  communicating  about  the  task  at  hand? 

How  successful  have  team  members  been  with  their  roles? 


Send 


New  probe:  |" 

Add  I  Adda  Send 


|00:00:00 

Probe  Response  Timeout:  | 

! — 

(seconds) 

Send 

^  track  of  time  and  specify  probe 
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Facilitator  Screenshot 


Critical  Thinking  Assessment  Tool 


Participants 
Participant  One 
Participant  Two 


7  1 
G  - 
5  - 
4  - 
3  - 


Probe  Result 


Comments 

1)  These  are  comments  for  this  probe. 

2)  These  are  more  comments  from  the  other  participant. 
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Facilitator  Screenshot 


Critical  Thinking  Assessment  Tool 


Participants 


Participant  One 
Participant  Two 


/tnuBun 


Comments 


1 )  These  are  comments  for  the  second  probe. 

2)  These  are  more  comments  for  the  second  probe. 


viotis  pbe  response;! 
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Facilitator  Screenshot 


Critical  Thinking  Assessment  Tool 
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Workspaces  in  a  Distributed  Team 
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Team  Member  Screenshot 


Mission  Planning  Collaborative  Tool 


File  Edit  Window  Help 


©I  fil^SlQl  ®|® 
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Opportunity  to  Rate  and  Comment 


gl  Mission  Planning  Collaborative  Tool 


File  Edit  Window  Help 


m  sBlAli’Ol 


jnjxj 


b,.CCL  -  Evaluate 


||  This  is  the  context  question  for  the  comment 


Comments: 


Canncel  | 


05:23  PM 
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